Can ChatGPT draft a research article? An example of population-level vaccine effectiveness analysis

We reflect on our experiences of using Generative Pre-trained Transformer ChatGPT, a chatbot launched by OpenAI in November 2022, to draft a research article. We aim to demonstrate how ChatGPT could help researchers to accelerate drafting their papers. We created a simulated data set of 100 000 health care workers with varying ages, Body Mass Index (BMI), and risk profiles. Simulation data allow analysts to test statistical analysis techniques, such as machine-learning based approaches, without compromising patient privacy. Infections were simulated with a randomized probability of hospitalisation. A subset of these fictitious people was vaccinated with a fictional vaccine that reduced this probability of hospitalisation after infection. We then used ChatGPT to help us decide how to handle the simulated data in order to determine vaccine effectiveness and draft a related research paper. AI-based language models in data analysis and scientific writing are an area of growing interest, and this exemplar analysis aims to contribute to the understanding of how ChatGPT can be used to facilitate these tasks.

W e reflect on our experiences of using Generative Pre-trained Transformer ChatGPT, a chatbot launched by OpenAI in November 2022, to draft a research article. We aim to demonstrate how ChatGPT could help researchers to accelerate drafting their papers. We created a simulated data set of 100 000 health care workers with varying ages, Body Mass Index (BMI), and risk profiles. Simulation data allow analysts to test statistical analysis techniques, such as machine-learning based approaches, without compromising patient privacy. Infections were simulated with a randomized probability of hospitalisation. A subset of these fictitious people was vaccinated with a fictional vaccine that reduced this probability of hospitalisation after infection. We then used ChatGPT to help us decide how to handle the simulated data in order to determine vaccine effectiveness and draft a related research paper. AI-based language models in data analysis and scientific writing are an area of growing interest, and this exemplar analysis aims to contribute to the understanding of how ChatGPT can be used to facilitate these tasks.
The introduction of ChatGPT (Generative Pre-trained Transformer) [1] in November 2022 by OpenAI captured public attention world-wide. It was instantly recognized as an entirely new level of service that artificial intelligence (AI) can offer to humanity in searching for information, answers, or solutions online. It is based on both supervised and reinforcement learning techniques [2], using human trainers for both approaches. It became popular among millions of users for its detailed, structured responses to questions on literally any subject, although it has been recognized that its responses can still be inaccurate or misleading [3].
At the time of writing this editorial, the PubMed -an inclusive global database of biomedical literature -registered the first eight publication that mentioned ChatGPT in some context. The first posed the question if the programmers would be replaced by ChatGPT [4], and the next two were concerned about undermining all essay assignments [5,6]. The next three worried if this would do more help or harm to nursing education, prescribing drugs, and medical students' education [7][8][9]. Finally, it's been shown that ChatGPT's abstracts of research papers can pass plagiarism detectors with 100% originality score and that scientists can no longer be sure if the abstract is an original work by their fellow scientists, or if it is generated by ChatGPT [10]. This raised important questions about authorship and publication ethics [11].

OUR EXPERIENCE OF USING CHATGPT TO DRAFT A RESEARCH ARTICLE
For this paper, we tried using ChatGPT to draft a research article, aiming to explore if ChatGPT could help researchers draft their papers. We created a simulated data set of 100 000 health care workers with varying ages, body mass index, and risk profiles. Simulation data allow analysts to test new and modern statistical analysis techniques, such as machine-learning based approaches, without compromising patient privacy. Infections (in this particular case, we used COVID-19 as an example) were simulated with a randomized probability of hospitalisation. A subset of the fictitious people was vaccinated with a fictional vaccine that reduced this probability of hospitalisation after infection. We then used ChatGPT to help us decide what to do with the simulated data to determine vaccine effectiveness and draft a related research paper.

Describing the data
First, we described the simulated data set to ChatGPT. The following is the first conversation between one of the authors and ChatGPT.
CM: I have a data set consisting of a cohort of 100 000 health care workers, some are vaccinated, and some are not. We also have data on if they were hospitalised or not due to COVID-19.
ChatGPT: Based on this data set, you can perform statistical analysis to determine if there is a correlation between vaccination status and hospitalization due to COVID-19. This could be done by calculating the proportion of vaccinated individuals who were hospitalized, and comparing it to the proportion of unvaccinated individuals who were hospitalized. You can also use techniques such as χ 2 test or logistic regression to analyze the relationship between the two variables. Additionally, it would be useful to consider other factors such as age, pre-existing conditions, and occupation, which may also affect the risk of hospitalization. Another statistical analysis that you could perform is the survival analysis. This is particularly useful if you also have information on the time between vaccination and hospitalization, and it allows you to estimate the hazard ratio of hospitalization for vaccinated individuals compared to unvaccinated individuals while accounting for the time-to-event aspect of the data.
Clearly, once we introduced ChatGPT to the nature of our data set, it did an excellent job in explaining what options in terms of study design were available to researchers and what would the outcomes mean.

Writing code
In the next step, we provided ChatGPT with a description of the data set (Table 1), in terms of the names of the columns in the database, what each column contained, and what was the nature of the data (e.g. integer, character, etc.). We then asked it to generate code using the programming language R to perform the survival analysis and calculate hazard ratios, as it had suggested itself.
The code contained a few mistakes. We were able to feed back this information to ChatGPT and allow it to self-correct. A good example of this is that ChatGPT initially wanted to use the vaccination date as a covariate. However, it then suggested to use a variable for the vaccination status at the time of hospitalisation instead. Finally, we made the following suggestion and ChatGPT was able to write simple, suitable code:

CM: A vaccine can take at least 14 days for a host to mount a response to the vaccine and provide protection against hospitalisation. So maybe we should account for this in the model?
ChatGPT: Yes, you are correct that it can take time for an individual to develop immunity after being vaccinated, so it would be appropriate to account for this in your survival analysis. One way to do this is to use the time since vaccination (v1_date) as a separate variable in the model, in addition to the vaccination status variable.
Here is an example of how you can adjust the survival analysis code to include the time since vaccination: With some additional prompts and some corrections, ChatGPT was also able to generate the code to extract hazards ratios with 95% confidence intervals (CI) from the fitted Cox model and display these in a Forest plot (Figure 1). By running the code written by ChatG-PT on the simulated data set, an adjusted hazards ratio for those who had been simulated to be vaccinated was calculated to be 0.48 (95% confidence interval (CI) = 0.28-0.86) -a value consistent with the parameter used in the simulation to reduce the hazard of hospitalisation for those who were vaccinated.

Writing the abstract
After ChatGPT was told what the numerical value of the hazard ratio (and its CIs) were, it was asked to begin writing the main contents of the scientific paper:  "vaccination" AND "healthcare workers" AND "hospitalization" "COVID-19" AND "vaccination" AND "healthcare workers" "healthcare worker" AND "vaccination" AND "COVID-19" Upon a few more attempts and queries, ChatGPT was able to do some literature search and find the entire list of references for the manuscript.

ChatGPT: I apologize, I am not able to provide you with a list of recent papers on vaccine-effectiveness as my knowledge is cut-off on 2021 and I don't have access to the internet to look for recent papers.
However, here is a list of papers on vaccine-effectiveness that were published between 2020-2021, that could be used as references for your paper:

"The effectiveness of the Pfizer-BioNTech COVID-19 vaccine" by Baden et al. (2020) in New England
Journal of Medicine.

Writing the introduction
The next step was to generate an introduction for the paper, and it was asked to justify some of its clams:

ChatGPT:
Sure, here is a revised version of the introduction section: Introduction: The These references provide evidence that vaccination is a key strategy to reduce the risk of severe illness and death among health care workers and it is recommended by international organizations such as the CDC and WHO.

Writing the methods
As this was all taking part in the same ChatGPT session, it was able to retain information that had been previously given as prompts, such as the nature and size of the data set.

ChatGPT:
Methods: This study is a retrospective observational study that utilizes a simulated data set of 100 000 health care workers from a made-up country. Inclusion criteria for the study were health care workers aged 18 years or older who were employed at the time of the study, and exclusion criteria were health care workers who were not employed or were not in the country during the study period. Information on vaccination status, age, sex, body mass index, number of comorbidities, and date of hospitalization were collected from electronic health records and administrative records.
Statistical analysis was conducted using Cox proportional hazards models to estimate hazard ratios for hospitalization among vaccinated individuals compared to unvaccinated individuals, while controlling for potential confounders such as age, sex, body mass index, and number of comorbidities. The time-varying effect of vaccination on hospitalization was also examined by including the time since vaccination in the model. All analyses were conducted using the R programming language.
This study was deemed exempt from institutional review board review as it was based on simulated data. However, we made sure that the simulated data were similar to real-world data and it was not possible to identify any individual from the data set.

Writing the results and discussion
Finally, ChatGPT was asked to write results and discussion section in concordance to the STROBE guidelines: It would now be possible to continue asking ChatGPT targeted questions to amend this brief discussion, such as asking it to explain the key strengths and limitations of the study, compare its results to the other similar studies that have already been published, highlight the remaining controversies and open questions, and eventually recommend further research that would be required to resolve them.
We believe that this viewpoint demonstrates that ChatGPT could become a great help to researchers worldwide in designing their studies, conducting analyses and drafting their research articles. Through this, it should be expected to accelerate the scientific progress. However, caution is required because it clearly needs human oversight in all stages and final input in order to guarantee accuracy or the reported results.
Its use in writing research papers clearly raises some ethical issues. First, it will be interesting if its frequent use leads to overly similar paragraphs and structure of many papers in the same field, leading to problems with plagiarism check. Then, our paper shows that references provided by ChatGPT cannot be trusted at the present time. An improvement will be needed in the future versions to address this issue [12].
An additional issue raised by the international press recently is that ChatGPT has been listed as co-author on several papers already, and scientific journals then moved quickly to ban its mention as a co-author [11]. One of the references in this editorial also named ChatGPT as a co-author [7]. The question whether co-authorship should also be assigned to ChatGPT if it drafted large parts of the paper will likely continue to be discussed. Its use should likely be acknowledged and explained at minimum. In a back-to-back editorial on this matter in the Journal of Global Health, we report on the latest guidance from WAME on writing articles using AI [13,14].
We also need to be careful about inferring too generally from a single case study that we present here. However, knowing that this is the first version of ChatGPT and realizing its capability to draft an entire research article based on the data set with which it has been presented suggests that we may be about to witness another revolutionary progress for science.